P2 HLT timing: check on all expected json files to declare success #2662

mmusich · 2026-01-21T20:19:24Z

Fixes #2658.
Title says it all, we need to check all (release-dependent) flavours after PR cms-sw/cmssw#49279

cmsbuild · 2026-01-21T20:19:48Z

A new Pull Request was created by @mmusich for branch master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-01-21T20:19:49Z

cms-bot internal usage

mmusich · 2026-01-21T20:20:22Z

test parameters:

enable = hlt_p2_timing

mmusich · 2026-01-21T20:20:28Z

@cmsbuild, please test

cmsbuild · 2026-01-21T22:11:58Z

-1

Failed Tests: HLTP2Timing
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f44e5/50795/summary.html
COMMIT: 522c9ac
CMSSW: CMSSW_16_1_X_2026-01-21-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2662/50795/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025513
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

cmsbuild · 2026-01-22T09:03:44Z

Pull request #2662 was updated.

mmusich · 2026-01-22T09:04:27Z

@cmsbuild, please test

cmsbuild · 2026-01-22T11:07:21Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f44e5/50805/summary.html
COMMIT: dafde32
CMSSW: CMSSW_16_1_X_2026-01-21-2300/el8_amd64_gcc13
Additional Tests: HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2662/50805/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025516
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

mmusich · 2026-01-22T15:21:55Z

@smuzaffar can we merge this? it will help catching failures in tests that went unnoticed so far.

smuzaffar · 2026-01-22T17:33:06Z

pr_testing/run-pr-hlt-p2-timing.sh

+    "$WORKSPACE/rundir/Phase2Timing_resources.json"
+    "$WORKSPACE/rundir/Phase2Timing_OnCPU_resources.json"
+    "$WORKSPACE/rundir/Phase2Timing_resources_NGT.json"
+)


@mmusich , do old releases also generate these files? Note that we are running these tests for CMSSW_14_1_X and above.

For IBs tests, we are uploading Phase2Timing_resources*.json files , should we change it to upload mv Phase2Timing_*.json ?

@smuzaffar, thank you for reviewing.

do old releases also generate these files? Note that we are running these tests for CMSSW_14_1_X and above.

This is the current situation:

CMSSW_16_1_X: generates all three files

CMSSW_16_0_X: generates all three files

CMSSW_15_1_X: generates Phase2Timing_resources.json and Phase2Timing_resources_NGT.json

CMSSW_15_0_X (and below): generates Phase2Timing_resources.json

how do you suggest to support a different number of files? I'd like to be fully explicit on the expected name of the json file: do we have means from the bash script to know the release version?

For IBs tests, we are uploading Phase2Timing_resources*.json files , should we change it to upload mv Phase2Timing_*.json ?

No. Actually thanks for pointing this out. This explains the mystery at cms-sw/cmssw#49279 (comment) !
I will rather change the name of the output json in order to match the expected pattern.
By the way can you show me which file is responsible for generating the IB tests?

Thanks!

looking at wha'ts done at cmssw-pr-test-config would:

CMSSW_VER=$CMSSW_VERSION required_files=( "$WORKSPACE/rundir/Phase2Timing_resources.json" ) if [ "$CMSSW_VER" -ge 1501 ]; then required_files+=( "$WORKSPACE/rundir/Phase2Timing_resources_NGT.json" ) fi if [ "$CMSSW_VER" -ge 1600 ]; then required_files+=( "$WORKSPACE/rundir/Phase2Timing_OnCPU_resources.json" ) fi

work?

By the way can you show me which file is responsible for generating the IB tests?

@mmusich , that code is in jenkins job itself (https://cmssdt.cern.ch/jenkins/job/ib-run-hlt-p2-timing/configure)

echo "git cms-addpkg HLTrigger/Configuration" >> $WORKSPACE/run.sh echo "cd $WORKSPACE/$RELEASE_FORMAT/src/HLTrigger/Configuration/python/HLT_75e33/test" >> $WORKSPACE/run.sh echo "timeout $TIMEOUT ./runHLTTiming.sh > $WORKSPACE/upload/run.log 2>&1" >> $WORKSPACE/run.sh echo "mv Phase2Timing_resources*.json $WORKSPACE/upload" >> $WORKSPACE/run.sh

looking at wha'ts done at cmssw-pr-test-config would:

some this should work, we can always run/request PR tests for old release cycles to check if it works properly or not. Also instead of converting CMSSW_VERSION (CMSSW_16_0_X) to CMSSW_VER (1600) in p2 timing script, I would suggest to update cms-bot/pr_testing/setup-pr-test-env.sh always have CMSSW_VERSION_NUMER and use it in p2 timign script (note that setup-pr-test-env.sh is sources in all PR test jobs to may be other test can also make use of it)

I would suggest to update cms-bot/pr_testing/setup-pr-test-env.sh always have CMSSW_VERSION_NUMER and use it in p2 timign script (note that setup-pr-test-env.sh is sources in all PR test jobs to may be other test can also make use of it)

just to be sure I understand, the proposal is to add in cms-bot/pr_testing/setup-pr-test-env.sh this line:

CMSSW_VERSION_NUMBER=$CMSSW_VERSION`

and then structure the if / else as

if [ "$CMSSW_VERSION_NUMBER" -ge 1501 ]; then ...

?
If you confirm I'll provide a commit shortly.

simple CMSSW_VERSION_NUMBER=$CMSSW_VERSION is not enough. You need to add something like the following or feel free to implement it with simpler logic :-) All we need is to convert major and minor version of release cycle in. to two digits and concatenate then to get CMSSW_VERSION_NUMBER

CMSSW_VERSION_NUMBER=$CMSSW_VERSION [ "${CMSSW_VERSION_NUMBER}" != "" ] || CMSSW_VERSION_NUMBER=${RELEASE_FORMAT} CMSSW_MAJOR=0 CMSSW_MINOR=0 if [ "${CMSSW_VERSION_NUMBER}" != "" ] ; then CMSSW_MAJOR=$(echo ${CMSSW_VER} | cut -d_ -f2) CMSSW_MINOR=$(echo ${CMSSW_VER} | cut -d_ -f3) fi export CMSSW_VERSION_NUMBER=$(echo x0${CMSSW_MAJOR}x0${CMSSW_MINOR} | sed -r -e 's|x[0]*([0-9][0-9])|\1|g;s|^0||')

Tried to implement it in 84487f1.

As in IBs tests uploading Phase2Timing_resources*.json, see cms-sw/cms-bot#2662 (comment)

pr_testing/run-pr-hlt-p2-timing.sh

- list of expected files is release dependent - changed setup-pr-test-env.sh to provide from the environment CMSSW version number

cmsbuild · 2026-01-23T10:18:22Z

Pull request #2662 was updated.

mmusich · 2026-01-23T10:19:46Z

pr_testing/run-pr-hlt-p2-timing.sh

+
+if [ "$CMSSW_VERSION_NUMBER" -ge 1600 ]; then
+    required_files+=(
+        "$WORKSPACE/rundir/Phase2Timing_resources_OnCPU.json"


notice that for this work, it will require to have cms-sw/cmssw#49887 integrated (and backported).

mmusich · 2026-01-23T10:20:26Z

test parameters:

enable = hlt_p2_timing
pull_request = runHLTTiming: add checking for errors in logs cmssw#49887

As in IBs tests uploading Phase2Timing_resources*.json, see cms-sw/cms-bot#2662 (comment)

smuzaffar · 2026-01-23T13:37:16Z

please test

cmsbuild · 2026-01-23T15:42:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f44e5/50837/summary.html
COMMIT: fc64764
CMSSW: CMSSW_16_1_X_2026-01-23-1100/el8_amd64_gcc13
Additional Tests: HLT_P2_TIMING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2662/50837/install.sh to create a dev area with all the needed externals and cmssw changes.

HLT P2 Timing: chart

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 52
DQMHistoTests: Total histograms compared: 4025536
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4025510
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
Checked 222 log files, 193 edm output root files, 52 DQM output files
TriggerResults: no differences found

smuzaffar · 2026-01-23T15:45:30Z

test parameters:

enable = hlt_p2_timing

pr_testing/setup-pr-test-env.sh

Co-authored-by: Malik Shahzad Muzaffar <shahzad.malik.muzaffar@cern.ch>

cmsbuild · 2026-01-23T15:49:33Z

Pull request #2662 was updated.

smuzaffar · 2026-01-23T15:51:09Z

please test with cms-sw/cmssw#49887

smuzaffar · 2026-01-23T15:51:42Z

please test for CMSSW_16_0_X

smuzaffar · 2026-01-23T15:52:02Z

please test for CMSSW_15_1_X

mmusich · 2026-01-23T15:52:37Z

please test for CMSSW_16_0_X

I think for this we need to test with cms-sw/cmssw#49909

smuzaffar · 2026-01-23T17:35:54Z

now CMSSW_VERSION_NUMBER=1501 is properly set

mmusich mentioned this pull request Jan 21, 2026

runHLTTiming: add checking for errors in logs cms-sw/cmssw#49887

Merged

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jan 21, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 21, 2026

cmsbuild added tests-rejected and removed tests-started labels Jan 21, 2026

mmusich force-pushed the mm_check_all_jsons branch from 522c9ac to dafde32 Compare January 22, 2026 09:03

cmsbuild added tests-pending and removed tests-rejected labels Jan 22, 2026

cmsbuild added tests-started and removed tests-pending labels Jan 22, 2026

cmsbuild added tests-approved and removed tests-started labels Jan 22, 2026

smuzaffar reviewed Jan 22, 2026

View reviewed changes

mmusich added a commit to mmusich/cmssw that referenced this pull request Jan 23, 2026

Fix name of timing json for CPU-only check.

8ce0a6c

As in IBs tests uploading Phase2Timing_resources*.json, see cms-sw/cms-bot#2662 (comment)

smuzaffar reviewed Jan 23, 2026

View reviewed changes

pr_testing/run-pr-hlt-p2-timing.sh Show resolved Hide resolved

mmusich added 2 commits January 23, 2026 11:14

P2 HLT timing: check on all expected json files to declare success

84487f1

- list of expected files is release dependent - changed setup-pr-test-env.sh to provide from the environment CMSSW version number

provide logs for phase HLT timing jobs

fc64764

mmusich commented Jan 23, 2026

View reviewed changes

cmsbuild added the requires-external label Jan 23, 2026

mmusich changed the title ~~P2 HLT timing: check on all 3 expected json files to declare success~~ P2 HLT timing: check on all expected json files to declare success Jan 23, 2026

mmusich mentioned this pull request Jan 23, 2026

getting logs out of hlt timing tests #2658

Open

mmusich added a commit to mmusich/cmssw that referenced this pull request Jan 23, 2026

Fix name of timing json for CPU-only check.

5a4b514

As in IBs tests uploading Phase2Timing_resources*.json, see cms-sw/cms-bot#2662 (comment)

mmusich mentioned this pull request Jan 23, 2026

[16.0.X] runHLTTiming: add checking for errors in logs cms-sw/cmssw#49909

Open

cmsbuild added tests-started and removed tests-pending labels Jan 23, 2026

cmsbuild added tests-approved and removed tests-started labels Jan 23, 2026

cmsbuild removed the requires-external label Jan 23, 2026

smuzaffar reviewed Jan 23, 2026

View reviewed changes

pr_testing/setup-pr-test-env.sh Outdated Show resolved Hide resolved

fix typ.

0e43b52

Co-authored-by: Malik Shahzad Muzaffar <shahzad.malik.muzaffar@cern.ch>

cmsbuild added tests-pending and removed tests-approved labels Jan 23, 2026

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 23, 2026

cmsbuild removed the requires-external label Jan 23, 2026

P2 HLT timing: check on all expected json files to declare success #2662

Are you sure you want to change the base?

P2 HLT timing: check on all expected json files to declare success #2662

Conversation

mmusich commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jan 21, 2026

Uh oh!

cmsbuild commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented Jan 21, 2026

Uh oh!

mmusich commented Jan 21, 2026

Uh oh!

cmsbuild commented Jan 21, 2026

Comparison Summary

Uh oh!

cmsbuild commented Jan 22, 2026

Uh oh!

mmusich commented Jan 22, 2026

Uh oh!

cmsbuild commented Jan 22, 2026

Comparison Summary

Uh oh!

mmusich commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cmsbuild commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmusich commented Jan 23, 2026

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

cmsbuild commented Jan 23, 2026

Comparison Summary

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

Uh oh!

cmsbuild commented Jan 23, 2026

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

mmusich commented Jan 23, 2026

Uh oh!

smuzaffar commented Jan 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mmusich commented Jan 21, 2026 •

edited

Loading

cmsbuild commented Jan 21, 2026 •

edited

Loading